This disclosure relates generally to the field of image processing. More particularly, but not by way of limitation, this disclosure relates to compensating for unwanted image distortions resulting from the so-called “rolling shutter” effect caused by certain complementary-metal-oxide-semiconductor (CMOS) sensors during video image capture operations.
Today, many personal electronic devices come equipped with digital image sensors that are capable of capturing video composed of a sequence of images. Exemplary personal electronic device of this sort include, but are not limited to, mobile telephones, personal digital assistants, portable music and video players and portable computer systems such as laptop, notebook and tablet computers. Many lower cost, high resolution cameras such as those utilized in compact, portable personal electronic devices are equipped with low-cost, low-power, CMOS sensors that can potentially geometrically distort captured images if there is movement of the device or the object being imaged while the CMOS sensor is capturing the scene.
An image sensor converts photons into electrons, thus converting optical images into electrical signals. Typically, image sensors may be either a charge-coupled device (CCD) or a CMOS. A CMOS sensor, unlike the CCD sensor, does not expose the entire sensor array at the same time since it cannot store and hold all of the individual pixel charges for the entire sensor array. Instead, CMOS sensors employ a so-called “rolling shutter” technique, wherein each row or scan line of the sensor array is exposed at different times, read out sequentially (e.g., from the top of the sensor to the bottom of the sensor), and then merged together to form a single image.
As long as the camera device and the object being imaged are stationary with each other, the output image typically does not include any geometric distortions caused by the “rolling shutter.” However, if there is relative movement horizontally or vertically between the image sensor and the object being imaged, the output image may potentially be distorted or temporally sheared, as shown in FIG. 1. This type of distortion is one example of what will be referred to herein as the “rolling shutter effect.” Resulting image frames and video sequences suffering from rolling shutter distortions are often aesthetically unpleasing and unwanted, as they do not accurately represent the scene being captured. Further, rolling shutter artifacts can worsen with high resolution images and high frame rates, e.g., 1080p images captured at 30 frames per second.
Some video capture devices now include “on board” motion sensors, i.e., positional sensors (e.g., accelerometers and/or gyrometers), which may be used to assist in various device functions. For example, some devices may use gyrometer data to aid in image stabilization by appropriately adjusting the device's lens and/or sensor mechanism before an image or frame is captured. Once captured, however, the image is retained as part of the video sequence without substantial modification. This approach is not, however, feasible for many devices incorporating video capture capability. For example, at this time, it is generally considered infeasible to provide movable lens mechanisms and the like in such small form factor devices.
Accordingly, there is a need for techniques to reduce the effects of rolling shutter distortion during image and video capture in devices utilizing CMOS or other non-CCD image sensors. By employing appropriate perspective transformations to captured image data based on timestamped information gathered from positional sensors in communication with the image capture device, more efficient image processing techniques may be employed to reduce the effects of rolling shutter distortion. By using novel motion compensation techniques, informed by hardware motion sensors, such as positional sensors, in communication with an image capture device, a robust rolling shutter reduction system may be employed, even in situations where reliably reducing rolling shutter distortion effects was previously thought to be impossible from either computational and/or power consumption standpoints.